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FINAL  REPORT: 

FACTORS  INFLUENCING  SIMULATOR  TRAINING  EFFECTIVENESS 
IN  THE  U.S.  AIR  FORCE 


Background 

It  has  been  noted  for  several  years  that  military  services  tend  to  use 
•aircraft  simulators  somewhat  less  extensively  than  do  certain  civilian 
organizations  in  achieving  flight  training  goals.  In  commenting  upon  this 
situation,  the  Comptroller  General  of  the  United  States  stated  the  following: 

"Although  the  military  services  have  taken  some  steps  to  increase  simu- 
lator development  and  use,  there  continues  to  be  resistance  to  substitution 
of  simulation  for  flying  time.  More  emphasis  on  the  use  of  simulators  is 
required  if  flying  time  is  to  be  reduced  and  if  the  services  are  to  effec- 
tively use  simulator  technology." 

The  referenced  report  goes  on  to  identify  a number  of  factors,  princi- 
pally factors  related  to  the  management  of  simulator  training  and  attitudes 
toward  such  training,  which  are  believed  to  impede  more  extensive  use  of 
simulators.  Factors  Identified  include  regulations  which  emphasize  aircraft 
rather  than  simulator  training,  inadequate  instructor  training,  failure  to 
use  simulator  capabilities  fully,  and  poor  simulator  maintenance.  The 
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influence  of  such  factors  upon  trainer  use  has  been  documented  elsewhere. 
Additional  factors  related  to  the  design  of  the  simulators  themselves  and 
of  the  training  programs  in  which  the  simulators  are  employed  have  been 

^Comptroller  General  of  the  United  States.  Department  of  Defense  Use 
of  Flight  Simo  hi  tors — Accomplishments.  Probl ems , and  Possible  Savings. 
Washington,  D.C.  : GAO  Code  952050,  June  l't,  1975. 

Smode,  A.F.  , Hall,  F..R.,  & Meyer,  D.E.  An  Assessment  of  Research 
Relevant  to  Pilot  Training,.  AMRL  TR  bb- 196.  • Aerospace  Medical  Research 
Lahorntor  ion,  Wr  I glit -Pal  I orson  AFH,  Ohio,  lPt-hi. 
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identified  as  factors  which  Impede  simulator  use  In  other  reports.  ’ . Still 
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other  factors  influence  the  acceptance  of  such  devices  and  may  indirectly 
impede  their  use. 

It  is  possible,  however,  to  use  simulators  extensively  while  at  the 
same  time  to  use  them  ineffectively.  In  one  study  of  Army  devices,  for 
example,  it  was  found  that  the  extensive  use  of  a particular  device  added 
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cost,  but  no  training  value,  to  an  already  expensive  pilot  training  program. 
Clearly,  the  intent  of  all  simulator  users  is  to  achieve  effective  training 
rather  than  merely  to  use  training  equipment.  Studies  of  trainer  use,  such 
as  those  cited  here,  tend  to  make  the  tacit  assumption  that  users  of  simu- 
lators will  assure  that  the  devices  are  used  effectively,  but  such  is  not 
always  the  case.  Effective  use  of  simulators  is  critical,  however,  if  the 
OSD-announced  goal  of  a 25%  reduction  in  flight  training  by  1980  is  to  be 
achieved. 

Because  of  its  extensive  investment  in  simulators  and  its  reliance 
upon  these  devices  for  significant  portions  of  the  training  of  its  aircrews, 
the  Air  Force  is  especially  aware  of  the  need  for  simulator  training  to  be 
of  maximum  possible  effectiveness.  Toward  that  end,  the  Human  Resources 
Laboratory  at  Williams  Air  Force  Base  has  devoted  major  efforts  and  resources 
to  research  into  simulator  design  and  utilization  factors,  and  major  Air 
Force  simulator  users,  such  ns  the  Tactical  Air  Command,  have  established 
projects  and  project  offices  to  evaluate  the  effectiveness  of  ongoing 

Meyer,  D.E.,  Flcxm.in,  R.E.,  Van  Gundy,  E.A.  , Killian,  D.C.,  & Lanahan, 

C . J . A Study  of  Simulator  Capabilities  in  an  Operational  Training  Program . 
AMRL  TR  67-14.  Aerospace  Medical  Research  Laboratories,  Wright-Patterson 
AFB,  Ohio,  1967. 
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simulator  training  programs.  While  reliable  data  are  not  immediately 
available,  it  is  believed  that  major  savings  in  dollars  and  fuel  could 
result  from  even  relatively  small  increases  in  the  effectiveness  of  current 
simulator  training  programs  in  the  Air  Force. 

Interest  in  the  effective  use  of  simulators  is  not  limited  to  the 
Department  of  Defense  and  its  pilot  training  agencies.  For  example,  the 
Principal  Investigator  for  the  project  reported  herein  was  asked  by  the 
Royal  Aeronautical  Society  (RAS)  to  prepare  a paper  on  the  subject  of 

simulator  training  effectiveness.  The  title  of  the  paper  is  "Some  Factors 
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Influencing  Transfer  of  Simulator  Training,"  and  its  content  discusses 
information  derived  from  a survey  of  the  research  and  training  literature 
and  from  the  author's  experience  related  to  influences  upon  simulator 
training  effectiveness.  An  objective  in  the  preparation  of  the  paper  was 
to  systematize  the  available  information  so  that  it  could  be  more  readily 
used  to  identify  factors  which  impede  the  effectiveness  of  simulator 
training,  and  thus  enable  operational  personnel  to  increase  simulator 
training  effectiveness  through  the  appropriate  manipulation  of  those  factors. 

The  RAS  paper  did  not  treat  the  sublet  as  comprehensively  as  might 
be  desired,  however.  Tts  preparation  was  constrained  in  two  ways: 

(1)  a limit  of  18  pages  in  length  was  imposed  by  the  Society;  and  (2)  the 
budget  under  which  the  paper  was  prepared  did  not  permit  full  treatment 
of  the  topics  or  investigation  of  ongoing  simulator  training  programs 
In  order  to  identify  additional  factors  which  were  influencing  the  effec- 
tiveness of  current  simulator  training  programs.  The  literature  survey 
did  not  exhaust  the  available  sources,  and  several  topics  which  were  iden- 
tified as  relevant  to  the  survey  could  not  be  included  because  of  the  space 
limitation.  A more  thorough  treatment  was  considered  desirable  of  factors 
Influencing  simulator  training  effectiveness  and  of  the  manner  in  which 
simulator  training  effectiveness  is  determined,  the  Letter  topic  being  a 
secondary  subject  discussed  in  the  RAS  paper. 

The  project  summarized  in  this  Final  Report  consisted  of  efforts  to 
expand  the  RAS  paper  beyond  its  original  limited  scope,  and  to  include 
material  about  t tie  effectiveness  of  Air  Force 's  i inula  tor  training.  Specific 
Interest  was  expressed  hv  Air  Force  personnel  with  whom  the  original  paper 
was  discussed  in  determining  the  extent  to  which  the  effectiveness  of  Air 
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Force  simulator  training  might  be  increased  through  modified  utilization 
practices  and  in  identifying  needs  for  research  which  could  lead  to  more 
effective  simulator  training.  The  present  project  was  in  part  a response 
to  these  expressed  interests  with  respect  to  expanding  the  scope  of  the 
paper  and  making  it  more  relevant  to  Air  Force  simulator  traing  per  se, 
as  well  as  to  identifying  needs  for  increased  simulator  training  wherever 
possible. 

Approach 

The  conduct  of  the  research  consisted  of  three  principal  activities: 
(1)  literature  surveys;  (2)  visits  to  selected  Air  Force  simulator  training 
facilities;  and  (3)  reporting.  In  general,  the  literature  survey  was  con- 
ducted before  the  visits  were  made  to  Air  Force  facilities,  although  there 
was  some  temporal  overlap  in  these  two  activities. 

Literature  Survey.  The  literature  surveys  concentrated  upon  the 
HumRRO  Pensacola  Office  Technical  Library.  Because  of  previous  HumRRO 
studies  of  flight  training  and  simulation,  this  library  had  good  coverage 
of  the  subject  area  of  concern  in  the  present  study.  As  information  rele- 
vant to  the  objectives  of  the  research  was  found  during  the  literature 
survey,  it  was  integrated  with  information  contained  in  the  RAS  paper.  In 
addition,  information  gathered  during  the  visits  to  ongoing  simulator 
training  activities  pertinent  to  the  literature  being  reviewed  was  incor- 
porated into  the  revision  of  the  RAS  paper  as  appropriate. 

Visits  to  Simulator  Training  Facilities.  In  order  to  gather  informa- 
tion concerning  Air  Force  simulator  effectiveness,  visits  were  made  to 
ten  simulator  training  activities  at  nine  Air  Force  bases.  During  these 
visits,  ongoing  simulator  training  programs  were  observed,  and  interviews 
were  conducted  with  a sample  of  the  instructors,  students,  and  training 
program  administrative  or  supervisory  personnel  associated  with  such  train- 
ing. The  purpose  of  the  observations  and  interviews  was  to  secure  infor- 
mation to  include  in  the  project  reports. 

Prior  to  vLsiting  Air  Force  simulator  training  facilities,  interview 
guides  were  prepared.  These  guLdos  served  as  checklists  to  assure  that 
the  time  spent  at  each  simulator  training  facility  was  productive.  The 
guides  addressed  the  available  data  concerning  the  effectiveness  of  the 


A 


simulator  training  under  study  and  the  manner  in  which  those  data  were 
obtained;  the  possible  influence  of  factors  which  the  literature  survey 
suggested  are  important  upon  the  simulator  training  being  conducted;  and 
factors  which  were  perceived  as  Influential  by  the  researcher  and/or  by 
local  training  and  supervisory  personnel  but  were  not  identified  during  the 
literature  survey. 

Table  1 identifies  the  Air  Force  bases  visited  during  this  research 
and  the  simulators  involved  in  the  training  activities  investigated  at  each. 
These  bases  and  simulators  were  selected  to  represent  Air  Force  simulator 
training  activities  beyond  the  Undergraduate  Pilot  Training  (UPT)  level. 

UPT  simulator  training  activities  were  not  investigated  during  the  present 
research  because  the  nature  of  UPT  simulator  training  is  expected  to  under- 
go significant  change  in  the  near  future  with  the  Introduction  of  new 
simulators  currently  being  procured  for  that  program. 

TABLE  1.  SIMULATOR  TRAINING  PROGRAMS  SURVEYED 


Air  Force 


Combat  Crew  Training 


Continuation  Training 


Aircraft 


Location 


Aircraft 


ADCOM 

F- 106 

Tyndall  AFB 

F- 106 

Castle  AFB 

MAC 

C-5 

Altus  AFB 

C-5 

Travis  AFB 

SAC 

B-52 

Carswell  AFB 

B-52 

Castle  AFB 

FB-  1 1 1 

Plattsburgh  AFB 

TAC 

F-4 

Luke  AFB 

F-4 

Eglin  AFB 

A- 7 

Dnvis-Monthnn  AFB 

The  simulator  training  programs  Included  In  the  survey  were  selected 
to  be  represent.! tive  of  Air  Force  aircraft  and  mission  types.  Both  Combat 
Crew  Training  (CCT)  and  f.ont  Imintion  Truinlng  (CT)  simulator  programs  were 
surveyed  for  the  Aerospace  Defense  Command  (ACCOM) , the  Military  Airlift 
Command  (MAC),  the  Strategic  Air  Command  (SAC),  and  the  Tactical  Air 
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Command  (TAC).  For  each  command,  the  same  simul.ilors/nlrcraft  were  surveyed 
with  respect  to  both  CCT  and  CT,  c.g.,  the  MAC  portion  of  the  survey  concen- 
trated upon  the  C-5  aircraft  for  both  CCT  and  CT  activities.  In  the  cases 
of  SAC  and  TAC,  the  survey  of  CCT  activities  included  an  additional  aircraft 
not  included  in  the  surveys  of  CT  activities.  This  was  done  because  of  the 
large  differences  in  aircraft  configuration  and  crewing  assignments,  compared 
with  the  other  aircraft  surveyed,  and  the  additional  aircraft  represented 
(l.e. , the  two-place  FB— ill  for  SAC  and  the  single-place  A-7  for  TAC).  It 
was  thought  that  including  these  additional  aircraft  might  yield  data  of 
interest  to  the  purposes  of  this  project  which  would  not  apply  to  the  other 
aircraft  Included  in  the  survey.  Thus,  the  survey  examined  six  CCT  and  four 
CT  programs  of  the  four  commands  located  at  nine  Air  Force  bases.  Six 
different  aircraft  were  simulated  by  the  devices  examined  during  the  survey. 
For  several  of  these  aircraft,  different  models  of  the  same  aircraft  were 
represented  by  these  simulators,  c.g.,  the  B-52D  and  the  B-52G. 

In  addition  to  the  interviews  conducted  at  the  locations  identified  , 
in  Table  1,  the  simulators  themselves  were  examined,  and  Air  Force  docu- 
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ments  describing  the  devices  and  the  manner  of  their  use  were  reviewed. 
Particular  attention  was  directed  to  seeking  documented  evidence  of  simula- 
tor training  effectiveness  at  each  location.  Where  no  documented  evidence 
of  the  effectiveness  of  simulator  training  could  ho  found,  personnel 
responsible  for  the  conduct  of  such  training  were  questioned  extensively 
in  an  attempt  to  determine  the  perceived  value,  quantitative  or  otherwise, 
of  such  training  and  the  bases  for  their  perceptions.  As  was  stated  earlier, 
no  attempts  wore  made  to  conduct  experimental  studies  to  determine  the 
effectiveness  of  simulator  training  activities.  — - - - 

Report lng.  The  results  of  the  research  program  were  presented  in  a 
Tcchn I ca 1 Report  ent it  1 ed , Some  Factors  Influencing  Air  Force  Simulator 
Training  Effectiveness,  March,  1977.  The  report  describes  the  project 
activities  leading  to  preparation  of  the  research  report  in  more  detail 
than  Is  contained  In  the  present  Final  Report.  . 

Findings  and  Kc common da t Ions 

The  principal  findings  and  recommendat Ions  of  the  research  project  are 
■tnmmar  I red  below: 


6 


bindings . Ten  simulator  training  effectiveness  study  designs  that 
have  been  used  in  studies  of  simulator  training  effectiveness  were  identi- 
fied and  described  in  terms  of  simple  models.  The  descriptions  of  these 
models  include  information  concerning  the  relative  value  of  each  with  res- 
pect to  the  relevance  and  objectivity  of  the  data  it  yields.  The  efforts 
by  the  Air  Force  to  validate  the  simulator  training  activities  surveyed  are 
described  in  relation  to  these  ten  study  design  models.  It  was  found  that 
the  programs  surveyed  had  not  been  subjected  to  formal  evaluation  studies 
that  would  establish  their  training  effectiveness  in  quantitative  terms. 

In  those  instances  in  which  attempts  had  been  made  to  validate  simulator 
training,  a tendency  was  noted  to  employ  study  models  that  were  based  upon 
subjective  opinions  rather  than  upon  objective  data  collected  during  trans- 
fer of  training  studies. 

A number  of  suspected  or  potential  factors  influencing  simulator 
training  effectiveness  were  identified  during  the  project.  These  factors 
include  simulator  design  for  training,  simulator  visual  display  fidelity, 
simulator  platform  motion  system  fidelity,  simulator  handling  characteris- 
tics, simulator  training  program  features,  simulator  trainee  and  instructor 
characteristics,  and  attitudes  and  expectations  toward  simulator  training. 

The  discussion  of  each  of  these  factors  reviews  relevant  literature  and 
Air  Force  simulator  and  training  system  design  features  and  training  prac- 
tices . 

The  available  information  concerning  the  influence  of  these  factors 
upon  simulator  training  effectiveness  was  found  to  be  quite  limited. 

Because  of  the  absence  of  objective  studies  validating  Air  Force  simulator 
training  effectiveness,  the  influence  of  factors  identified  during  the 
survey  upon  such  training  could  only  be  hypothesized.  Additionally,  defini- 
tive data  could  seldom  be  found  in  the  literature  reviewed  that  would  per- 
mit the  quantification  of  the.  influence  of  many  suspected  factors,  and 
methodological  problems  made  it  difficult  to  generalize  conclusions  from 
the  literature  to  the  Air  Force  programs  surveyed.  Therefore,  the  infor- 
mation presented  in  this  report  regarding  influences  upon  simulator  training 
effectiveness  In  some  instances  is  suggestive  rather  than  conclusive. 

Recommendations.  Principal  recommendations  are  as  follows: 
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— Increased  emphasis  should  be  placed  upon  validating  Air  Force  simulator 


training  activities,  employing  validation  study  design  models  that  emphasize 
objective  measurement  of  trainee  performance  in  operational  aircraft  against 
predetermined  performance  standards. 

— Research  should  be  undertaken  to  examine  simulator  design  considera- 
tions as  a function  of  specified  training  objectives  and  of  the  manner  in 
which  the  devices  are  to  be  used  to  achieve  those  objectives. 

— Research  should  be  undertaken  to  determine  the  cognitive  and  visual 
cues  essential  to  the  attainment  of  visual  training  objectives  and  to  find 
means  of  attaining  those  objectives  that  do  not  rely  exclusively  upon 
extra-cockpit  visual  simulation. 

— Research  should  be  undertaken  to  examine  separately  the  influences 
of  maneuver  and  disturbance  motion  cues,  with  particular  attention  to  an 
analysis  of  disturbance  motion  cues  in  relation  to  specific  training  objec- 
tives. 

— Reviews  of  Air  Force  simulator  training  activities  should  be  con- 
ducted to  identify  areas  in  which  better  use  could  be  made  of  available 
information  in  the  areas  of  human  learning  and  performance. 

Increased  emphasis  should  be  placed  upon  the  needs  of  individual 
trainees  in  the  development  and  administration  of  simulator  training 
programs. 

Research  should  be  undertaken  to  identify  the  instructor  skills  and 
techniques  needed  for  effective  and  efficient  simulator  training,  and 
training  programs  in  which  such  skills  and  techniques  could  be  developed 
should  be  provided  all  simulator  instructors. 

Existing  administrative  practices  related  to  simulator  training 
should  be  examined  to  assure  that  they  are  conducive  to  favorable  attitudes 
toward  simulator  training  and  to  the  effectiveness  of  that  training. 

Other  Project  Activities 

In  addition  to  the  principal  project  activity  described  above,  the 
Principal  Investigator  made  a presentation  at  the  38th  Military  Operations 
Research  Society  Symposium  as  a part  of  the  rpsearch  project.  The  presen- 
tation made  use  of  information  that  had  been  developed  during  the  surveys 
of  Air  Force  simulator  training  activities  described  above,  as  well  as 
information  developed  during  previous  research  sponsored  by  the  Army,  Navy 
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and  Coast  Guard.  A Technical  Report,  Some  Current  Problems  in  Simulator 
Design,  Testing  and  Use,  was  prepared  based  upon  the  symposium  presentation. 
That  Technical  Report  is  summarized  below: 

"This  report  is  concerned  with  the  general  problem  of  the  effectiveness 
of  simulator  training  and  reflects  information  developed  during  the  conduct 
of  aircraft  simulator  training  research  projects  sponsored  by  the  Air  Force, 

Army,  Navy,  and  Coast  Guard.  Problems  are  -identified  related  to  simulator 
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design,  testing  and  use  that  impact  simulator  training  effectiveness.  These 
problems  are  (1)  isolation  of  the  simulator  user  from  the  design  and  develop- 
ment process,  (2)  inattention  to  behavioral  and  training  models  during  that 
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process,  (3)  ignoring  training  considerations  during  simulator  tesing, 

(4)  inadequate  feedback  to  simulator  designers  concerning  simulator  training 
effectiveness,  (5)  inattention  to  techniques  of  simulator  training  that  differ 
from  techniques  of  aircraft  training,  (6)  inadequate  training  for  simulator 
instructors,  (7)  use  of  rate  of  simulator  utilization  as  an  index  of  its 
training  effectiveness,  and  (8)  inadequacies  of  simulator  training  cost 
* effectiveness  data." 

In  addition  to  the  above,  a paper  was  prepared  and  submitted  to  the 
Naval  Training  Equipment  Center  for  presentation  at  the  10th  NTEC/Industry 
Conference  to  be  held  in  the  fall  of  1977.  The  paper,  titled  "Platform 
Motion  and  Simulator  Training  Effectiveness,"  is  based  upon  information 
developed  during  the  current  research  project  related  to  the  influence  of 
motion  cues  upon  the  effectiveness  of  training  in  aircraft  simulators.  The 
paper  is  summarized  below: 

"Several  recent  studies  reported  that  simulator  motion  did  not  benefit 
subsequent  flight  performance.  Other  studies  have  reported  various  effects 
of  motion  upon  pilot  performance  in  simulators.  These  possibly  contradic- 
tory findings  are  examined  in  the  light  of  recent  distinctions  made  between 
maneuver  and  disturbance  motion.  Tine  studies  in  which  simulator  motion 
did  not  benefit  transfer  predominantly  employed  maneuver  motion  cues, 
whereas  the  other  group  of  studies  incorporated  more  disturbance  motion 
cues.  Pilot  reactions  to  simulator  motion  also  were  examined  in  terms  of 
maneuver  vs.  disturbance  motion,  and  it  was  hoted  that  judgments  of  the 
training  value  of  simulator  motion  were  related  to  the  maneuver-disturbance 
distinction.  It  is  concluded  that  maneuver  jnotion  may  be  of  little 
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potential  training  value,  under  many  circumstances,  and  data  necessary  to 
an  adequate  simulation  of  disturbance  motion  may  not  be  available.  An 
analysis  of  the  training  requirements  associated  with  disturbance  w m ie 
needed. " 

It  should  be  noted  that  this  paper  has  not  yet  been  accepted  by  the 
Conference  Program  Committee. 
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